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ABSTRACT 


The  goal  of  the  North  Atlantic  Treaty  Organization’s  (NATO)  Connected  Forces 
Initiative  (CFI)  is  to  optimize  the  ability  of  NATO  forces  to  operate  together,  through  technical 
interoperability  and  multi-national  training.  While  the  CFI  seeks  to  achieve  cross-national 
efficiencies,  the  functional  communities  within  the  member-nation  militaries  generally  focus  in 
improving  their  internal  efficiencies,  rather  than  considering  opportunities  to  obtain  cross 
functional  efficiencies.  In  some  cases,  this  is  due  to  cultural  differences  that  define  the  decision¬ 
making  drivers  and  values  of  the  two  communities.  The  primacy  of  prioritization  in  the 
operational  training  community,  versus  Cost- As- An-Independent- Variable  (CAIV)  in  the  testing 
community,  and  the  preference  for  complete  control  by  both  groups  are  identified  as  major 
challenge  to  conducting  integrated  activities.  Using  the  Best  Value  contracting  model  as 
precedence,  this  paper  proposes  a  decision-support  model  that  applies  a  marginal-cost 
comparison  approach  to  a  System-of-Systems-Analysis  (SOSA)  to  package  costs  and  benefits 
into  a  value-judgment  decision  framework  to  support  decision-making  for  integrated  testing  and 
training  events.  The  certification  of  multi-national  receiver  for  air-refueling  is  used  as  an 
example  case. 


Introduction 


The  North  Atlantic  Treaty  Organization’s  (NATO)  International  Security  Assistance 
Force  (ISAF)  mission  in  Afghanistan  is  projected  to  end  after  2014.  At  the  conclusion  of  this 
mission,  NATO  expects  to  shift  its  focus  from  current  operations  to  operational  readiness.  The 
Connected  Forces  Initiative  (CFI)  is  one  of  the  policy  constructs  that  implements  this  change  in 
focus.  The  CFI  calls  for  NATO  members  to,  “ensure  that  Allies  can  communicate,  train  and 
operate  together  effectively,  and  that  NATO  has  increasing  opportunities  to  validate  and  certify 
their  ability  to  do  so.”1 

At  the  national-level,  the  CFI’s  objective  of  enhancing  the  effectiveness  of  international 
forces  translates  into  reduced  defense  burdens  for  the  individual  member  states.  However,  at  the 
military/defense  department-level,  every  aspect  of  the  CFI  comes  with  a  cost.  Effectively 
operating  together  requires  interoperable  weapon-  and  training-systems,  effective  technical 
means  of  communication,  and  integrated  training.  Whether  the  system  is  as  complex  as  the 
Joint  Strike  Fighter  or  as  simple  as  a  standardized  rifle-round,  interoperable  systems  incur 
development,  test,  procurement,  and  sustainment  costs.3,4  For  many  of  the  expansion-members 
of  NATO,  this  represents  a  thorough  and  expensive  refitting  of  their  military  equipment  from 
Soviet-bloc  standards  to  NATO  standards.  For  example,  Poland’s  armed  forces  modernization 
plan  is  projected  to  cost  $30-40  billion  per  year,  for  the  next  10  years,  to  transition  ships, 
helicopters,  tanks,  aircraft,  personnel  carriers,  and  air  &  missile  defense  systems  to  modern, 
NATO  interoperable,  systems.5  Communicating  effectively  may  be  even  more  challenging  in 
the  long  run,  because  it  has  a  material  and  a  human  component.  NATO  forces  must  have  both 
technically  interoperable,  secure  communications  systems  and  personnel  who  are  proficient  at 
communicating,  often  in  non-native  languages,  under  the  pressure  of  combat  conditions.  While 


the  technical  gap  can  be  closed  within  a  generation  of  equipment  updates,  personnel  must 
interact  frequently  to  develop  new  personnel  and  maintain  their  proficiency.6  Finally,  simply 
conducting  military  training  is  an  expensive  endeavor,  largely  due  to  the  costs  of  energy. 

Dorothy  Robyn,  the  Deputy  Under  Secretary  of  Defense  for  Installations  &  Environment,  has 
identified  the  US  Department  of  Defense  as  the  nation’s  largest  energy  consumer,  with  over  $11 
billion  spent  on  operational  energy,  i.e.  fuel  for  tanks,  ships,  planes,  and  generators,  in  201 1 
alone.7  So,  fulfilling  the  CFI’s  objective  of  increasing  the  opportunities  to  train,  directly 
translates  to  increased  costs. 

However,  the  CFI  is  pressing  for  more  resources  at  a  time  when  the  member  governments 
of  NATO  are  pushing  for  even  lower  defense  expenditures.  In  response  to  the  worst  economic 
slump  since  the  Second  World  War,  governments  are  restricting  or  reducing  their  defense 
spending,  some  sharply.  For  example,  between  2009  and  2011,  the  European  NATO  members 
cut  $45  billion  from  the  defense  expenditures.9  By  the  same  token,  the  2013  US  budget  is 
projected  to  be  reduced  by  $454  billion  between  2013  and  2021. 10  While  emerging  threats  could 
reverse  this  trend,  the  general  direction  of  defense  spending  is  currently  downward,  and  will 
likely  remain  that  way  for  the  foreseeable  future. 

Large  government  institutions,  like  defense  departments,  have  a  limited  number  of 
strategic-level  options  for  moving  forward  with  diminishing  resources.  First,  they  can  reduce 
their  capabilities,  i.e.  simply  do  less  in  order  to  live  within  the  constraints.  Second,  they  can 
spread  their  material  investment  out  over  a  longer  period.  This  reduces  the  annual  costs  of 
programs,  but  usually  increases  the  total  costs.  Third,  they  can  defer  activities  and  programs  in 
order  to  keep  certain  elements  of  their  forces  at  full  capability  while  allowing  others  to  diminish. 
Finally,  they  can  seek  efficiencies  within  their  existing  programs  in  order  to  “do  more,  with 


less.”  Traditionally,  this  is  the  preferred  approach  of  governments,  because  it  is  the  easiest 
option  politically. 

It  is  taken  as  an  assumption  that  NATO,  in  general,  will  pursue  the  fourth  option  in 
response  to  increasing  budgetary  pressure.  In  the  natural  flow  of  responsibility  within 
bureaucracies,  the  task  of  actually  finding  efficiencies  to  implement  the  CFI,  while 
simultaneously  preserving  capability,  will  fall  to  the  military.  Within  the  NATO  construct,  this 
makes  the  US  dual-hatted  Supreme  Allied  Commander  Europe  (SACEUR)/Commander,  United 
States  European  Command  (CDR  USEUCOM)  especially  important,  since  the  US  currently 
contributes  approximately  75-percent  of  NATO  military  expenditures.11  Yet,  he  has  only  a 
limited  range  of  options  that  are  under  his  direct  control.  The  balancing  decision  between 
development,  procurement,  and  operations  will  be  made  at  higher  levels  of  the  services, 
departments,  and  government.  So,  his  attention  will  naturally  fall  upon  the  optimization  of  the 
operations  under  his  direct  control.  However,  while  the  bulk  of  the  procurement  community  is 
not  in  a  position  to  provide  any  form  of  synergy,  there  are  certain  situations  where  the  testing 
community  may  be  able  to.  As  a  result,  there  is  an  under-leveraged  opportunity  to  increase 
overall  efficiency  and  effectiveness,  at  the  macro-level,  by  integrating  testing  and  training 
events.  But,  this  is  only  occasionally  acted  upon  because  it  reduces  the  internal  efficiency  of 
each  domain. 

The  prevailing  communities-of-practice,  in  both  disciplines,  steer  their  members  to  a 
perspective  that  excludes  impartial  consideration  of  the  net  efficiencies  that  could  be  gained. 

The  initial  perspective  of  both  groups  is  a  client/service-provider  model,  wherein  the  client  is 
trying  to  minimize  his  costs  and  the  service  provider  is  trying  to  maximize  his  “profit.”  As  a 
result,  the  entering  operational  perspective  is  usually  that  the  test  center  should  pay  for 


absolutely  everything  that  is  used,  disrupted,  leveraged,  occupied,  or  expended  for  the  test. 
Essentially,  the  testing  organizations  should  pay  for  the  complete  opportunity  cost  of  supporting 
the  test,  even  if  supporting  the  test  is  transparent  to  the  assets  involved,  i.e.  they  would  have  been 
there  anyway.  Conversely,  the  Test  Director  will  naturally  only  want  to  pay  for  the  actual  costs 
of  the  activities  that  were  specifically  required  for  their  test  to  minimize  his  or  her  expenditures 
and  preserve  that  funding  for  other  activities.  This  is  a  natural  position  for  both  organizations  to 
adopt,  they  are  separate  entities,  with  distinctly  different  responsibilities,  and  are  both  resource 
constrained.  However,  it  is  not  a  conducive  mindset  to  explore  opportunities  to  increase 
collective  efficiency  or  effectiveness. 

Before  proceeding  with  the  discussion  of  collective  efficiency  and  effectiveness,  it  is 
necessary  to  distinguish  the  types  of  testing  and  training  in  question.  Not  all  testing  is 
appropriate  for  a  training  environment.  The  Measure-of-Performance  (MOP)  oriented  activities 
of  Developmental  Testing  and  Evaluation  (DT&E)  must  be  performed  in  a  completely  controlled 
environment  in  order  to  isolate  the  independent  and  dependent  variables.  By  the  same  token,  the 
measures-of-Effectiveness  (MOE)  based  Initial  Operational  Testing  and  Evaluation  (OT&E) 
must  be  completed  prior  to  the  certification  of  training  products,  technical  orders,  and  operations 
manuals  which  are  necessary  to  produce  Initial  Qualification  Training  (IQT)  programs.  So,  the 
tests  in  question  are  best  characterized  as  US  Major  Command  (MAJCOM)  Force  Development 
Evaluation  (FDE)  and  post-operational  Certification  tests,  such  as  the  verification  of  technically 
compatible  systems  for  Air-to-Air  Refueling.  ’  While  testing  focused  on  national-systems 
could  be  tested  in  the  context  of  a  NATO  environment,  activities  related  to  alliance 
interoperability  will  likely  be  preferred,  due  to  the  collective  Return-On-Investment  (ROI)  of 
their  accomplishment.  The  types  of  training  in  question  are  similarly  restricted.  The  trade-space 


under  discussion  is  only  relevant  for  the  final  mission  certification  or  readiness  exercises  of 
otherwise  fully  qualified  personnel.  In  other  words,  the  operational  exercise. 

Why  is  Macro-efficiency  not  a  Default  Answer? 

So,  why  do  organizations  within  the  same  service,  government,  or  at  least  industry,  not 
pursue  collective  perfonnance  as  a  default?  This  phenomenon  is  largely  a  function  of  how  their 
performance  is  measured.  The  training  and  testing  communities  have  different  policies, 
priorities,  and  measures  of  success.  Further,  the  resource  streams  for  these  activities  are,  at  least 
for  the  US  military,  separately  budgeted  and  managed.  As  a  result,  both  communities  have 
developed  specialized  processes  and  procedures  to  internally  satisfy  their  requirements  in 
accordance  with  the  criteria  by  which  they  are  judged.  Shifting  the  focus  of  their  optimization 
efforts  to  a  higher  level  may  be  contrary  to  the  performance  incentive  system. 

The  policies  that  define  training  requirements,  procedures,  and  practices  can  be  traced 
from  individual  career-field  guidance,  e.g.  pilots,  up  through  the  National  Defense  Strategy 
(NDS).  The  Department  of  Defense  (DoD)  uses  a  process  known  as  Global  Force  Management 
(GFM)  to  assign  and  assess  forces  to  the  Combatant  Commands  (COCOMs)  to  fulfill  their 
warfighting  mission.  The  foundation  of  this  process  is  the  assessment  of  the  readiness  of  forces 
to  fulfill  those  requirements.14  Training  is  a  measured  aspect  of  readiness  at  all  organizational 
levels  throughout  the  process.  The  Chairman  of  the  Joint  Chiefs  of  Staff  (CJCS)  provides 
guidance  and  policy  for  training,  which  flows  down  through  various  implementation  directives 
and  policy  documents  to  the  services. 15  For  the  USAF,  the  capstone  policy  directive  for 
readiness  is  AFPD  10-2  Readiness,  with  an  emphasis  on  reporting.16  Operational  organizations 
are  measured  against  both  baseline  standards  and  each  other.  This  combination  of  preparing  for 


warfighting  with  performance  openly  published  within  the  service  and  joint  community  means 
that  readiness  and  training  are  only  prioritized  below  actual  mission  execution. 

In  comparison  to  the  primacy  of  training,  testing  is  just  one  component  of  a  subordinate 
priority  of  equipping  the  force.  While  testing  will  ultimately  lead  to  the  fielding  of  new  or 
improved  equipment,  which  will  eventually  enhance  a  unit’s  capabilities,  any  time  or  resources 
diverted  to  this  objective  potentially  comes  at  the  expense  of  readiness.  Since  readiness  is  both  a 
higher  priority  and  the  measure  of  success  for  operational  units,  there  are  few  near-term 
incentives  to  supporting  integrated  testing  activities.  Additionally,  there  is  no  logical  upper  limit 
on  training  resource  requirements.  Training  can  and  will  consume  all  time  and  resources 
allocated  to  it.  This  creates  challenges  for  integrating  testing  and  training  events.  Training 
events  completed  are  significantly  more  valuable  to  trainers  than  test  events  completed. 

The  policy  and  guidance  for  military  testing  can  also  trace  a  path  to  the  National  Defense 
Strategy.  However,  it  is  constructed  along  functional,  rather  than  command,  lines  and  is  oriented 
towards  executive  decision-making,  rather  than  force  presentation.  The  top-level  document  that 
establishes  policy,  processes,  and  measures  of  success  for  testing  is  the  Department  of  Defense 
Instruction  (DODI)  5000.02,  “Operation  of  the  Defense  Acquisition  System”,  under  the  authority 
of  the  Secretary  of  Defense. 17  While  testers  are  independent  from  the  development  and 
procurement  centers,  they  are  still  part  of  the  acquisitions  community  and  are  governed  by  the 
overarching  guidance  of  DODI  5000.01,  which  directs  a  Total  Systems  approach,  which 
balances  system  performance,  operational  effectiveness  and  suitability,  survivability,  safety,  and 
affordability.  As  a  result,  the  measure  of  success  of  a  test  is  whether  it  was  rigorous  and 
complete  enough  to  accurately  assess  the  system  and  inform  decision-making. 


Testing  then,  like  the  larger  acquisitions  process,  is  about  balancing  time  and  resources 
against  a  myriad  of  specific  test  objectives  in  order  to  ensure  that  the  most  effective  test  possible 
is  conducted.  The  key  objective  during  test  design  is  well  paraphrased  from  the  Air  Force 
Operational  Test  and  Evaluation  AFOTEC  Manual,  “to  ensure  an  operationally  and  technically 
adequate,  credible  and  sufficient  test  where  limitations  and  mitigation  plans  are  clearly 
identified.”19  So,  while  the  basic  policy  of  testing  calls  for  balancing,  versus  strict  prioritization, 
the  execution  of  the  test,  as  designed,  is  a  tester’s  ultimate  objective. 

However,  the  test  community  faces  a  significant  barrier  to  integrated  activities  that  arises 
from  the  methods  of  testing,  rather  than  policy.  The  scientific  nature  of  testing,  and  the  need  to 
know  what  was  measured,  leads  testers  to  conduct  progressive,  controlled  events  that  build  in 
complexity  after  verifying  the  preceding  foundational  steps.  This  accomplishes  two  important 
tasks.  First,  it  manages  the  risks  to  personnel  and  material.  Second,  it  helps  to  isolate  the 
specific  conditions  and  causes  of  any  failures  that  occur.  The  tools  and  theory  of  testing  revolve 
around  concepts  such  a  Design  of  Experiment  (DOE),  which  focus  on  optimizing  test  efficiency 
and  effectiveness  with  limited  test  runs."  The  result  of  these  processes  is  a  highly  optimized  test 
plan  that  minimizes  the  duration,  and  consequently  the  cost,  of  a  test  while  simultaneously 
maintaining  a  controlled  and  safe  test  environment.  Consequently,  it  is  the  de-optimization  of 
the  test  process  and  the  associated  loss  of  control  of  the  test  environment  that  pose  the  greatest 
challenge  to  integrated  testing. 

In  addition  to  the  difference  in  culture  and  priorities  that  are  shaped  by  policy,  there  is  a 
higher-order  challenge  to  collective  efficiency  within  military  establishments,  which  is  one  of  the 
features  that  distinguishes  military  organizations  from  commercial  ones.  The  interactions 
between  military  organizations  are  dictated  by  priorities,  rather  than  contractual  agreements. 


Each  commander  is  responsible  and  accountable  for  the  execution  of  his  or  her  missions  in  order 
of  their  importance.  The  need  to  fulfill  these  obligations  exceeds  the  requirements  to  honor 
previous  commitments  or  external  agreements.  Any  memorandums  of  Agreement  (MOAs)  that 
are  signed  by  operational  agencies  will  usually  include  a  higher-priority  exception  clause  to  the 
support  that  they  are  committing  to  provide.  But,  even  if  it  were  not,  an  operational  commander 
who  breaks  an  agreement  to  support  a  testing  activity  due  to  higher  operational  priorities  will  not 
be  censured  by  their  superiors  in  any  way  because  they  would  be  acting  in  accordance  with  the 
larger  institution’s  values.  The  impact  to  the  test  community  is,  quite  literally,  not  their  problem. 
So,  while  the  concept  of  military  priority  is  itself  rational  and  acceptable,  it  is  an  impediment  to 
integrated  testing. 

This  poses  a  serious  challenge  to  those  considering  collective  efficiency  and 
effectiveness.  While  service  members  may  trust  each  other  with  their  lives,  service  agencies 
don’t  trust  each  other  with  their  money.  There  is  significant  historical  evidence  to  support  this 
distrust.  In  the  US  Air  Force,  fluctuation  in  fuel  prices  can  create  major  “bills”  during  a  fiscal 
year  (FY),  which  must  be  paid  for  from  within  the  service.  In  FY12,  the  USAF  faced  a  $  1 .3 
billion  funding  shortfall  for  fuel,  due  to  price  increases.  For  perspective,  that  equates  to  about  a 
$  1 .00  per  barrel  difference  in  the  price  of  oil.21  The  Air  Force  has  limited  options  for  mitigating 
this  type  of  expense:  fly-less,  repurpose  O&M  funds,  or  move  money  from  investment  accounts. 
In  order  to  fulfill  operational  requirements,  the  bulk  of  this  bill  was  paid  for  from  multi-year 
investment  accounts,  i.e.  development  and  procurement  funds.  By  the  same  token,  maintaining 
readiness  continues  to  be  a  top  priority.  As  the  Air  Force  Chief  of  Staff,  General  Mark  Welsh 
stated  in  regards  to  the  FY14  budget,  “Any  relief  in  the  FY14  budget  will  go  into  Readiness. 
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Readiness,  it's  at  the  top  of  our  payback  list.”  So,  the  testing  community  is  constantly  at  risk  of 
losing  support  or  pooled  funds  to  the  higher-prioritization  of  the  operational  community. 

The  lynchpin  to  moving  forward  in  this  area  is  higher-level  prioritization  of  the  integrated 
event.  Within  military  hierarchies,  or  even  international  bodies  such  as  NATO,  there  are 
priority-setting  bodies  that  can  protect  the  execution  of  integrated  efforts  from  changing 
circumstances  and  ensure  that  resources  will  be  retained  to  their  original  purposes  if  the  events 
are  disrupted.  It  requires  a  very  high-level  of  staff  coordination  to  accomplish  this,  because  the 
testing  and  training  chains-of-command  are  generally  segregated  until  the  highest  levels  of 
defense  establishments. 

While  the  challenges  of  culture,  policies,  and  priorities  between  these  two  communities 
frequently  inhibit  close  cooperation,  there  is  precedence  for  an  alternative  approach.  The 
foundation  of  one  alternative  is  found  within  the  framework  of  defense  contracting.  Of  course, 
applying  this  approach  requires  the  reimagining  of  the  operational  decision-making  process  as  a 
contracting  process. 

The  operational  community  generally  approaches  problem-solving  by  considering 
requirements,  then  balancing  priorities  and  risks.  The  apportionment  of  forces  or  levels-of-effort 
that  are  assigned  to  achieve  any  particular  objective  is  the  product  of  the  minimum  effort 
required  to  accomplish  the  mission,  modified  by  the  commander’s  tolerance  for  risk,  and  the 
demands  of  competing  priorities.  For  operational  missions,  planners  will  not  generally  consider 
monetary  costs.  However,  training  activities  typically  are  capped  by  a  maximum  monetary 
expenditure.  But,  the  planning-problem  is  usually  framed  as  an  optimization  problem,  i.e. 
maximizing  the  number  and  priority  of  training  objectives  that  can  be  accomplished  with  the 
available  resources.  This  approach  is  analogous  to  the  contracting  source-selection  process 


known  as  Lowest-Priced  Technically  Acceptable  (LPTA).  This  is  the  classic  government 
approach  of  going  with  the  lowest  bidder. 

An  alternative  approach,  which  has  been  formally  documented  and  implemented  for 
contract  source-selection,  is  known  as  the  Trade-off  Method  for  achieving  best  value.  These 
source-selections  are  commonly  referred  to  simply  as  “Best  Value.”  A  Best  Value  award  allows 
the  Source  Selection  Authority  (SSA)  to  make  tradeoffs  between  technical  performance  and 
other  considerations,  against  price.  This  allows  the  SSA  to  select  the  best  overall  value  to  the 
government,  rather  than  selecting  the  lowest-priced  option  that  fulfills  the  minimum 
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requirements.  This  model  establishes  a  framework  for  planners  to  consider  the  value  of  a 
longer  training  period,  versus  the  total  cost  of  an  event. 

Decision-Support  Model 

Armed  with  an  understanding  of  the  challenges  and  a  construct  for  explaining  an 
alternative  approach,  it  is  now  possible  to  propose  a  decision-support  model  for  capturing  the 
value  of  integrated  testing  and  training  events  that  is  both  informative  and  streamlined  enough 
for  use  by  the  Staff  Officers  that  drive  executive  decision-making  in  military  hierarchies.  This 
model  will  address  a  series  of  questions.  The  first  key  question  is  related  to  testing  conditions. 
Can  the  test  be  accomplished  outside  of  the  exercise’s  events  or  does  it  require  conditions  that 
only  exist  in  the  midst  of  the  exercise’s  activities?  If  the  test  must  be  conducted  within  the 
context  of  the  larger  training  event,  how  disruptive  is  it?  In  other  words,  what  are  the 
opportunity  costs  of  conducting  integrated  testing  and  training?  The  second  key  question  is 
related  to  monetary  costs.  What  are  the  costs  of  deploying  and  sustaining  the  training  and  test 
assets,  broken  out  with  sufficient  fidelity  to  allow  the  comparison  of  alternate  scenarios?  The 
final  question  is  predicated  on  the  answers  to  the  first  two.  Can  the  inclusion  of  the  test  event 


produce  enough  savings  to  “buy  back”  the  opportunity  and  literal  costs  of  fulfdling  its 
requirements? 

The  first  step  in  developing  a  decision  model  for  merging  testing  and  training  events  is  to 
establish  whether  the  activities  are  merely  combined  or  truly  integrated.  This  is  an  important 
distinction  that  leads  to  the  subsequent  questions  regarding  how  disruptive  the  events  are  and  the 
associated  opportunity  costs.  Combined  events  will  generally  have  greater  planning  flexibility 
than  integrated  events. 

A  combined  testing/training  event  is  one  wherein  the  testing  program  can  be  completely 
deconflicted  from  the  exercise  events  and  merely  requires  access  to  personnel  and  equipment,  in 
combination,  at  some  point  during  the  event.  A  combined  testing/training  activity  can  be 
accomplished  prior  to,  in  between,  or  after  the  exercise’s  events  are  accomplished.  For  example, 
the  certification  of  some  military  transport  aircraft,  requires  that  fully  equipped  paratroopers  be 
timed  while  exiting  the  aircraft  in  a  ground  emergency  scenario.  This  test  could  be  accomplished 
on  a  non-interference  basis  within  an  exercise,  even  though  it  requires  the  dedicated  use  of  the 
assets  and  personnel. 

On  the  other  hand,  an  integrated  test  event  must  be  accomplished  within  the  context  of 
the  larger  exercise.  If  the  test  event  were  to  be  conducted  as  a  stand-alone  event,  creating  the 
relevant  test  environment  would  strongly  resemble  an  exercise.  For  example,  certifying  the 
effectiveness  of  a  communications  data-bridge  that  connects  multiple  national  and  service 
platforms,  under  operational  conditions,  would  require  the  same  variety  of  assets  and  participants 
as  a  command  and  control  exercise.  It  is  the  complexity  of  these  events  that  actually  creates  the 
potential  for  return-on-investment. 


If  the  testing  and  training  activities  must  be  integrated,  then  the  second  step  is  to 
determine  how  integrated  they  are.  In  today’s  highly  integrated  battlefield,  particularly  for 
NATO  forces  which  rely  on  their  integration  and  communications  to  be  a  force  multiplier,  it  is 
easy  to  declare  that  everything  is  interconnected.  This  is  true  enough,  but  the  degree  of 
interconnection  is  critical  for  informing  decision-making.  Otherwise,  it  tempts  commanders  to 
oppose  the  imposition  of  test  activities  on  their  exercises  and  to  attempt  to  bill  the  costs  of  the 
entire  endeavor  to  the  test  agency.  Meanwhile,  the  probability  that  the  entire  exercise  will  enter 
a  holding  pattern  around  the  test  is  actually  remote. 

Conducting  a  top-level  System-of-Systems  Analysis  (SOSA)  will  clarify  and  document 
which  elements  of  the  test  and  training  forces  are  actually  integrated  and  subsequently,  how  the 
shared  costs  of  integrated  or  independent  activities  should  be  calculated.  While  SOSA  may 
sound  like  an  engineering  or  sciences  methodology,  it  is  a  tool  that  is  already  in  the  military 
professional’s  toolbox.  The  US’s  Joint  Operation  Planning  Process  (JOPP)  uses  multiple 
iterations  of  SOSA  in  Operational  Design.  It  is  the  foundation  of  the  Political,  Military, 
Economic,  Social,  Infonnation,  and  Infrastructure  (PMESII)  analytical  framework  and  Critical 
Factors  Analysis.24  The  depth  of  JOPP  processes  also  model  appropriate  limits  for  the  depth  of 
analysis  that  is  required.  These  planning  processes  break  down  systems  three  steps  away  from 
their  objective.  This  is  consistent  with  Complexity  Theory’s  assertions  that  one  to  two  degrees 
of  freedom  is  sufficient  to  predict  system  performance.  Therefore,  this  model  proposes  a  three¬ 
tiered  analysis  of  affected  systems.  This  should  enable  test  and  exercise  planners  to  conduct 
useful  thought-experiments  early  on  in  the  process  to  develop  an  understanding  of  the 
interactions  between  the  elements  of  the  activity  prior  to  consideration  of  the  additional  friction. 


The  first  tier-units  for  the  SOSA  are  those  that  are  directly  involved  in  the  testing 
activities.  For  example,  in  the  CFI,  aerial  refueling  certifications  are  a  significant  issue  because 
the  member  nations  have  different  processes  and  requirements  for  approving  their  aircraft  to 
refuel  from  different  platforms.  In  this  scenario,  the  first  tier  operational  units  are  the  AR 
platform(s)  and  the  receiving  aircraft.  For  these  first  tier  assets,  the  testers  will  likely  need 
complete  control  of  the  asset  and  to  dictate  how  it  operates  in  order  to  maintain  control  of  the 
“experiment”  which  is  the  test.  If  this  is  the  case,  then  the  full  cost  of  these  activities,  in  hours, 
should  be  levied  against  the  test  itself.  However,  if  formal  exercise  objectives  can  be  fulfilled  by 
the  testing  activity,  within  its  constraints,  then  the  fractional  costs  of  the  activity  should  be  pro¬ 
rated  against  the  test.  Conversely,  the  value  of  “simply  flying”  or  “operating  the  system”  should 
not  be  pro-rated. 

The  second  tier  units  are  the  supporting  and  associated  elements  of  the  exercise  that  are 
actively,  but  not  directly,  involved  in  the  immediate  tasks  and  are  required  for  the  testing  activity 
to  be  performed.  This  tier  of  SOSA  will  frequently  be  occupied  by  the  Command  and  Control 
(C2)  elements.  In  this  case,  principally  airspace  controllers  and  planners.  These  elements  should 
be  evaluated  to  determine  if  there  are  differential  costs,  compared  to  their  normal  operations.  In 
other  words,  if  an  element  performs  its  function  regardless  of  the  addition  of  the  testing 
activities,  its  fractional  costs  should  not  be  levied  against  the  test  for  the  decision  space.  For 
example,  ground  controllers  operate  24/7  to  control  and  deconflict  airspace,  whether  there  are 
any  aircraft  to  control  or  not.  On  the  other  hand,  if  an  airborne  control  asset  has  to  be  launched 
early  to  support  the  testing  activity,  its  fractional  costs  should  be  considered. 

The  third  and  final  tier  of  affected  units  are  the  down-stream  assets  that  are  affected  only 
by  the  opportunity  costs  of  lost  training  time  associated  with  the  testing  activity.  These  costs 


should  initially  be  accounted  for  in  training/flying  hours,  rather  than  monetary  units,  because  the 
payback  of  the  integrated  testing  is  an  extended  training  period,  which  will  also  initially  be 
considered  in  time  units.  Figure  1  provides  an  example  three-tier  SOSA  for  air  refueling 
certification  testing  in  the  context  of  a  larger  exercise. 
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Figure  1.  Air  Refueling  Certification  SOSA 

The  SOSA  analysis  itself  merely  identifies  the  relevant  components  of  the  exercise  that 
should  be  accounted  for.  The  degree  of  disruption  or  delay  experienced  by  each  element  must  be 
quantified,  as  well  as  possible,  and  described  in  an  accessible,  narrative  format.  This  phase  will 
rely  on  the  Staff  Officer’s  understanding  of  the  overall  scheme  of  the  exercise  and  consultation 
with  the  affected  units  and  staff  functionals  to  characterize  the  impact  of  injecting  testing 
activities.  Both  the  tester’s  requirements  and  the  flexibility  of  the  exercise  plan  must  be  well 
understood  in  order  to  assess  and  quantify  the  fractional  impacts  of  the  testing.  Figure  2  is  a 
notional  example  of  an  impacts  roll-up  table  based  upon  the  example  SOSA.  Once  again,  the 


objective  level  of  detail  is  that  which  is  sufficient  to  inform  strategic  decision-making,  not  to 
identify  completely  accurate  and  precise  impacts.  As  a  result  of  the  briefing-oriented  decision- 

environment,  the  credibility  of  this  type  of  information  may  rest  more  upon  the  accessibility  and 
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clarity  of  its  presentation,  than  its  completeness  and  rigor." 


Notional  Training  Impacts  of  Supporting  Integrated  Testing 

Element 

Tier 

Impact 

Air  Refueling  (AR)  Aircraft  (A/C) 

1 

Two  (2)  A/C  require  on-site  instrumentation  and  3.0  hours  in  contact  with  receiver 
aircraft,  for  5  training  days  to  conduct  dry  &  wet  certifications  in  NLT  0.5  hour  increments. 
Contiguous  days  not  required.  A/C  can  be  tasked  for  out-of-contact  support. 

Receiver  Aircraft 

1 

Progressive,  non-contiguous:  1-ship  (2-days),  2-ship  (2-days),  and  4-ship  (1-day)  packages 
must  be  available,  for  3.0  in  contact  hours  per  day  in  NLT 0.5  hour  increments,  same- 
sortie  time-restricted  training  is  taskable  outside  of  contact  hours. 

Receiver  Aircraft  Maintenance 

II 

Additional  Maintenance  assets  required  for  above-exercise  turn  rates  fortesting  and 
training  sorties. 

Air  Refueling  Aircraft  Maintenanci 

II 

Priority  of  Mx  over  other  assigned  AR  assets  to  ensure  availability  fortesting 

Airfield  Operations 

II 

No  impact  relative  to  exercise  operations 

Ground  Airspace  Control 

II 

No  impact  relative  to  exercise  operations 

Airborne  Airspace  Control 

II 

Modification  to  orbit  location  to  support  testing  activities,  reduced  high-intensity 
training  time  for  controllers  on  supporting  crew  during  sorties  supporting  testing. 

BLUE  Package  Aircraft 

III 

Increased  complexity  to  scheduling  and  routing,  loss  of  potential  for  1.5  hours  per  day  of 
full-package  training  (after  scheduling  mitigations).  Airspace  restrictions  in  testing  zone. 

RED  Package  Aircraft 

III 

Extension  of  training  day  duration,  no  change  training  to  hours/scenarios. 

Air  Defense  Artillery 

III 

Extension  of  training  day  duration,  no  change  training  to  hours/scenarios. 

JTAC  Training/Certification 

III 

Extension  of  training  day  duration,  no  change  training  to  hours/scenarios. 

Bottom  Line  Impact:  Integrated  testing  increases  daily  scheduling  complexity,  extends  the  daily  training  period,  reduce  the 
potential  for  full -package  training  by  1.5  hours  per  day,  and  reduce  training  intensity  for  some  elements. 

Figure  2.  Notional  Impacts  of  Testing  on  Training  Environment  from  SOSA 


With  the  impacts  of  integrating  events  in  hand,  the  next  step  in  the  process  of  defining  the 
cost-benefit  space  is  to  assess  the  financial  cost  of  the  activities.  Unfortunately,  one  of  the  most 
difficult  questions  for  a  government  agency  to  answer  is  “how  much  does  it  cost?”  Many 
taxpayers  find  this  frustrating  and  assume  that  this  is  due  to  the  incompetency  of  the  government 
bureaucracy.  But,  this  is  not  necessarily  the  case.  It  is  mostly  the  result  of  the  scale  and 
complexity  of  the  services,  personnel,  and  material  that  governments  buy,  which  are  not  easily 
divisible  or  chargeable  to  a  specific  system,  project,  or  individual.  Rest  assured,  major 
companies  have  the  exact  same  problem.  To  fully  calculate  the  relevant  costs  for  an  integrated 


testing  and  training  event,  known  as  Operations  and  Sustainment  (O&S)  costs,  the  participating 
agencies  should  use  the  Office  of  the  Secretary  of  Defense  (OSD)  Cost  Analyses  and 
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Improvement  Group’s  (CAIG)  level-2  Cost  Element  Structure  (CES),  provided  in  Figure  3. 

Of  course,  at  this  level  of  accounting,  the  cost  of  this  research  paper  is  well  over  $4,000.  Its 
costs  include  the  fractional  time  of  the  officer  who  wrote  it,  the  government  laptop  it  was  written 
on,  the  cost  of  the  bandwidth  used  during  research,  the  instructor’s  time  that  was  applied  to  the 
course,  etc. 


Second-Level  Cost  Estimating  Structure 
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5.  Continuing  System  Improvements 

5.1  Hardware  Modifications  or  Modernization 

5.2  Software  Maintenance  and  Modifications 

6.  Indirect  Support 

6.1  Installation  Support 

6.2  Personnel  Support 

6.2.1  Personnel  Administration 
(Personnel  Acquisition , 
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6.3  General  Training  and  Education 

6.3.1  Basic  &  Initial  Skill  Training 

6.3.2  Educational  Activities 


Figure  3.  OSD  Level-2  O&S  Cost  Breakout 


Fortunately,  it  is  not  necessary  to  calculate  the  exact  cost  of  the  testing  or  training 


activities  to  inform  the  overall  decision-making  process.  The  only  real  question  that  needs  to  be 
answered  is  how  much  test  funding  can  be  freed  by  integrating  the  events  and  what  can  those 
funds  buy  that  is  useful  to  the  exercise.  Perhaps  the  most  valuable  resource  to  an  exercise 


planner  is  time.  Ironically,  this  will  frequently  form  the  basis  of  arguments  against  integrated 
testing  and  training.  Yet,  it  can  be  one  of  the  easiest  things  for  test  funding  to  procure.  If  the 
operating  costs  and  time-requirements  to  accomplish  the  testing  and  training  activities  are  held 
constant,  then  marginal  costs  of  moving  assets  for  testing  can  be  compared  to  the  costs  of 
extending  personnel  in  place  to  increase  the  overall  duration  of  the  exercise. 

Depending  upon  the  specific  scenario  the  source  for  costing  data  will  vary.  But,  for  the 
example  air-refueling  certification  testing,  the  US  government  publishes  the  relevant  information 
publicly.  The  focus  on  differential  costs  to  support  decision-making,  rather  than  specific  costs 
for  direct  accounting  dramatically  reduces  the  information  that  is  required.  The  Under  Secretary 
of  Defense  (Comptroller)  publishes  reimbursable  rates  for  the  use  of  fixed-  and  rotary-wing 
aircraft,  military  personnel,  and  other  categories  of  assets  that  could  potentially  be  used  on  a 
reimbursable  basis  by  DoD,  Federal,  and  non-government  agencies.  Additionally,  US 
Transportation  Command  (USTRANSCOM)  publishes  additional  surface-,  maritime-,  and  air-lift 
rates."  These  rates  are  drawn  from  the  same  data-set  and  calculated  in  the  same  manner.  They 
are  also  focused  on  the  immediate  operating  costs  of  the  platforms,  not  their  long-term 
sustainment  or  total  cost  of  ownership.  These  numbers  are  appropriate  for  considerations  of 
short-term  usage,  which  is  the  question  at  hand,  rather  than  multi-year  budgeting  decisions. 

These  rates  will  define  the  cost  of  moving  assets  for  an  independent  test  event,  with  all  other 
testing  costs  held  constant. 

In  order  to  calculate  the  potential  benefit  to  the  exercise,  the  basic  cost  of  time  for  the 
event,  not  the  costs  of  activities,  must  be  determined.  This  will  vary  with  the  specific  nature  of 
the  training  event.  However,  since  the  personnel  costs  of  active  duty  military  and  federal 
civilians  are  budgeted  annually  and  centrally  controlled,  there  is  only  an  opportunity  cost  for 


their  time,  rather  than  a  specific  billable  cost  related  to  the  exercise.  Instead,  for  a  NATO 
exercise,  the  principal  costs  are  those  associated  with  dislocating  personnel  from  their  home 
stations  to  conduct  the  event.  For  the  purposes  of  this  decision-support  tool,  the  transportation 
costs  for  both  the  testing  and  training  team  will  be  considered  a  sunk  cost.  The  focus  is  on  the 
daily  rate  of  keeping  personnel  on  site.  The  Government  Service  Agency  (GSA)  and  the  State 
Department  maintain  world-wide  rates  for  lodging,  meals,  expenses,  and  incidentals  which  will 
characterize  the  cost  of  living  on  the  local  economy  as  a  visitor  and  are  centrally  accessible  via 

30 

the  Defense  Travel  Organizations. 

For  the  example  scenario,  re -positioning  two  air  refueling  assets  from  the  eastern  coast  of 
the  United  States  to  United  States  European  Command  (EUCOM)  to  conduct  certification  testing 
will  require  approximately  10  hours  of  flying  time.  The  Test  Director  would  likely  include  at 
least  one  day  of  set-up  margin  and  an  extra  day  for  a  make-up  test-event,  which  will  be  charged 
at  the  USTRANSCOM  Minimum  Activity  Rate  (MAR).  The  actual  flying  hours  for  the  test 
activities  are  not  considered  here,  as  they  will  be  executed  in  either  scenario.  Additionally,  the 
Test  Director  will  be  billed  for  the  re-positioning  flight  to  return  the  aircraft  to  its  home  base. 

This  will  cost  around  $14,000  per  flying  hour  and  incur  a  $28,000  per  day  MAR  fee  for  the 
margin-days.  This  yields  roughly  $672,000  in  billable  charges  just  to  position  primary  and  back¬ 
up  Air  Refueling  assets.  The  VOQ  Rates  are  roughly  $56.00  per  night,  assuming  on-base 
lodging,  and  per-diem  is  roughly  $130.00  per  day.  At  those  rates,  an  integrated  test  can  buy  an 
additional  3,612  man-days  of  on-site  time.  Perhaps  more  accessibly,  that  buys  a  1,200-person 
exercise,  three  extra  days  on  location  to  conduct  their  activities. 

The  last  step  of  this  process  is  to  finish  translating  the  economic  costs  and  benefits  into 
training  costs  and  benefits  to  match  the  priority  driven  decision-making  construct  of  operational 


commanders.  In  other  words,  what  can  the  extra  time  do  for  the  training  command?  In  this  case, 
it  can  enhance  the  effectiveness  of  the  training  by  increasing  the  debriefing  and  review  time 
between  events,  when  it  is  most  effective.  It  can  add  schedule  margin  for  environmental 
conditions  or  issues,  and  create  the  opportunity  to  conduct  other  training  that  does  not  expend 
additional  resources.  Figure  4  presents  the  outputs  of  this  model  in  one  ubiquitous  PowerPoint 
slide. 
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Conclusion 


Efficiencies  must  be  found  in  order  for  NATO  forces  to  modernize  and  enhance  their 


ability  to  function  together,  in  the  face  of  global  economic  challenges.  Yet,  while  NATO  as  an 


institution  is  well  aware  of  the  cultural  differences  between  its  member  nations,  it  may  be 
unaware  of  the  cultural  differences  within  the  militaries  themselves.  As  a  result,  significant 
opportunities  for  cross-domain  efficiencies  may  go  overlooked.  This  is  particularly  true  between 
the  testing  and  the  training  communities,  which  due  to  their  function-centric  views  and 
differences  in  measures  of  success  are  inclined  to  form  ineffective  relationships.  The  key  to 
bridging  that  gap  is  to  frame  decisions  to  conduct  an  integrated  events  in  terms  of  costs  and 
benefits  that  are  relevant  to  the  training  command  and  highlight  where  those  costs  and  benefits 
lie.  Simple  cost  comparisons  are  an  inadequate  tool  for  the  task.  This  paper  has  presented  a 
decision-support  model  that  applies  a  marginal-cost  comparison  approach  to  a  System-of- 
Systems-Analysis  (SOSA)  to  package  costs  and  benefits  into  a  value-judgment  decision 
framework  that  should  be  within  the  comfortable  skillset  of  a  typical  Staff  Officer.  This 
framework  may  ease  the  cross-community  mistrust  and  enable  NATO  forces  to  achieve 
enhanced  efficiency  and  effectiveness  in  the  2 lil  century. 


Endnotes 


I  NATO.  “The  Connected  Forces  Initiative.”  Available  on-line  from: 
http://www.nato.int/cps/en/natolive/topics  98527.htm.  Accessed  on  18  Mar  2014. 

"  NATO.  “The  Connected  Forces  Initiative.”  Available  on-line  from: 
http://www.nato.int/cps/en/natolive/topics  98527.htm.  Accessed  on  18  Mar  2014. 

3  NATO.  NATO  Standardization  Agreement  4172:  5.56mm  Ammunition  (Linked  or  Otherwise),  Ed  2,  Amd  2,  31 
January  1995. 

4  Joint  Strike  Fighter  Program  Office.  History.  Available  on-line  from:  http://www.isf.mil/historv/his  f35.htm. 
Accessed  on  28  March  2014. 

5  Andrew  A.  Michta,  Polish  hard  power:  Investing  in  the  Military  as  Europe  Cuts  Back,  (American  Enterprise 
Institute,  December  19,  2013)  Available  online  from:  http://www.aei.org/outlook/foreign-and-defense- 
policv/defense/nato/polish-hard-power-investing-in-the-militarv-as-europe-cuts-back/.  Accessed  on  6  April  2014. 

6  Benjamin  S.  Lambeth,  NATO's  Air  War  for  Kosovo:  a  Strategic  and  Operational  Assessment  (Santa  Monica,  CA: 
RAND  Corporation,  2001),  166-170. 

7  Pellerin,  Cheryl.  DOD  Gives  High  Priority  to  Saving  Energy.  American  Forces  Press  Service.  Available  on-line 
from:  http://www.defense.gov/news/newsarticle.aspx7kD65480.  Accessed  on  6  April  2014. 

8  NATO.  “Smart  Defense.”  Available  on-line  from:  http://www.nato.int/cps/en/natolive/topics  84268.htm.  Accessed 
on  18  Mar  2014. 

9  Rasmussen,  Anders  Fogh.  Building  Security  in  an  Age  of  Austerity.  Speech:  20 1 1  Munich  Security  Conference 
(February  2011).  Available  on-line  from:  http://www.nato.int/cps/en/natolive/opinions  70400.htm.  Accessed  on  6 
Apr  2014. 

10  Congressional  Budget  Office.  Estimated  Impact  of  Automatic  Budget  Enforcement  Procedures  Specified  in  the 
Budget  Control  Act.  Report,  12  September  2011.  Available  on-line  from:  http://www.cbo.gov/publication/42754. 
Accessed  on  6  April  2014. 

II  Rasmussen,  Anders  Fogh.  Building  Security  in  an  Age  of  Austerity.  Speech:  2011  Munich  Security  Conference 
(February  201 1).  Available  on-line  from:  http://www.nato.int/cps/en/natolive/opinions  70400.htm.  Accessed  on  6 
Apr  2014. 

12  Air  Force  Operational  Testing  and  Evaluation  Center  Manual  (AFOTECM)  99-101.  Operational  Test  Processes 
and  Procedures,  11  Oct  2012. 

13  NATO  Standard  ATP-3. 3.4.2.  Air-To-Air  Refueling  ATP-56,  Ed.  C  Ver.  1.  November  2013. 

14  Chairman  Joint  Chiefs  of  Staff  (CJCS)  Guide  340 ID.  CJCS  Guide  to  the  Chairman 's  Readiness  System.  (15  Nov 
2010),  22-23. 

15  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  3500.01G,  Joint  Training  Policy  for  the  Armed  Forces 
of  the  United  States,  (12  March  2013),  A1-A10. 

16  Air  Force  Policy  Directive  (AFPD)  10-2.  Readiness,  (6  Nov  2012),  1. 

17  Department  of  Defense  Instruction  (DODI)  5000.02,  Operation  of  the  Defense  Acquisition  System,  (25  November 
2013),  1-2. 

18  Department  of  Defense  Instruction  (DODI)  5000.01,  The  Defense  Acquisition  System,  (20  November  2007),  10. 

19  Air  Force  Operational  Testing  and  Evaluation  Center  (AFOTEC)  Manual  99-101,  Operational  Test  Processes  and 
Procedures,  (1 1  October  2012),  20. 

20  Air  Force  Operational  Testing  and  Evaluation  Center  (AFOTEC)  Manual  99-101,  Operational  Test  Processes  and 
Procedures,  (1 1  October  2012),  49. 

21  Gabe  Starosta,  “The  Air  Force’s  Fuel  Problem,”  Air  Force  Magazine,  July  2012.  Available  on-line  from: 
http://www.airforcemag.com/MagazineArchive/Pages/2012/Julv%202012/0712fi.iel.aspx.  Accessed  on  27  Mar 
2014. 

22  Welsh,  Mark  A.  III.  Press  Briefing  by  Acting  Secretary  of  the  Air  Force  Eric  Fanning  and  Air  Force  Chief  of  Staff 
General  Mark  A.  Welsh  III  on  the  State  of  the  Air  Force  in  the  Pentagon  Briefing  Room.  (13  Dec  2013). 
http://www.defense.gov/transcripts/transcript.  aspx?transcriptid=5344. 

23  Defense  Acquisition  Guidebook.  Chapter  14:  Acquisition  of  Services.  Available  on-line  from:  https://dag.dau.mil 
Accessed  on  27  Mar  2014. 

24  Joint  Publication  5.0,  Joint  Operation  Planning,  (11  August  2011),  IIL9-IIL24. 

25  Eric  Berlow,  Simplifying  Complexity,  TEDGlobal,  July  2010. 


26  Miriam  J.  Metzger  and  Andrew  J.  Flanagin,  Credibility  and  Trust  of  Information  in  Online  Environments:  The 
Use  of  Cognitive  Heuristic,  (Journal  of  Pragmatics,  Vol.  59,  2013,  pp.  210-220),  213-214. 

27  OSD  Cost  Analysis  Improvement  Group  (CAIG),  Operating  &  Support  ( O&S )  Cost-Estimating  Guide,  (Oct 
2007),  6.1-6.16. 

28  Office  of  the  Undersecretary  of  Defense  (Comptroller).  Financial  Management  Reports,  Department  of  Defense 
FY2014  Reimbursable  Rates. 

29  United  States  Transportation  Command.  Rates  and  Rules:  FY14  Airlift  Rates.  Available  from: 
http://www.transcom.mil/rates/fyl4Rates/.  Accessed  on  26  Mar  2014. 

30  Defense  Travel  Management  Office.  Per  Diem  Rates  Query.  Available  online  from: 
http://www.defensetravel.dod.mil/site/perdiemCalc.cfm.  Accessed  on  26  Mar  2014. 


BIBLIOGRAPHY 


Air  Force  Operational  Testing  and  Evaluation  Center  (AFOTEC)  Manual  99-101.  Operational 
Test  Processes  and  Procedures.  1 1  October  2012. 

Air  Force  Operational  Testing  and  Evaluation  Center  Manual  (AFOTECM)  99-101.  Operational 
Test  Processes  and  Procedures,  1 1  Oct  2012. 

Air  Force  Policy  Directive  (AFPD)  10-2.  Readiness,  6  Nov  2012. 

Berlow,  Eric.  Simplifying  Complexity.  TEDGlobal.  July  2010.  Available  online  from: 

http://www.ted.com/talks/eric  berlow  how  complexity  leads  to  simplicity.  Accessed 
on  26  Mar  2014. 

Chairman  Joint  Chiefs  of  Staff  (CJCS)  Guide  340  ID.  CJCS  Guide  to  the  Chairman ’s  Readiness 
System.  15  Nov  2010. 

Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  3500. 01G.  Joint  Training  Policy  for 
the  Armed  Forces  of  the  United  States.  12  March  2013. 

Congressional  Budget  Office.  Estimated  Impact  of  Automatic  Budget  Enforcement  Procedures 
Specified  in  the  Budget  Control  Act.  Report,  12  September  2011.  Available  on-line  from: 
http://www.cbo.gov/publication/42754.  Accessed  on  6  April  2014. 

Defense  Acquisition  Guidebook.  Chapter  14:  Acquisition  of  Services.  Available  on-line  from: 
https://dag.dau.mil  Accessed  on  27  Mar  2014. 

Defense  Travel  Management  Office.  Per  Diem  Rates  Query.  Available  online  from: 

http://www.defensetravel.dod.mil/site/perdiemCalc.cfm.  Accessed  on  26  Mar  2014. 

Department  of  Defense  Instruction  (DODI)  5000.01.  The  Defense  Acquisition  System.  20 
November  2007. 

Department  of  Defense  Instruction  (DODI)  5000.02.  Operation  of  the  Defense  Acquisition 
System.  25  November  2013. 

Goldwater-Nichols  Department  of  Defense  Reorganization  Act  of 1986.  Pub  Law  99-433.  99th 
Cong.,  1  Oct  1986. 

Joint  Publication  5.0.  Joint  Operation  Planning.  1 1  August  2011. 

Joint  Strike  Fighter  Program  Office.  History.  Available  on-line  from: 

http://www.isf.mil/history/his  f35.htm.  Accessed  on  28  March  2014. 


Lambeth,  Benjamin  S.  NATO’s  Air  War  for  Kosovo:  A  Strategic  and  Operational  Assessment. 
Santa  Monica,  CA:  RAND  Corporation,  2001. 


Metzger,  Miriam  J.  and  Andrew  J.  Flanagin.  Credibility  and  Trust  of  Information  in  Online 

Environments:  The  Use  of  Cognitive  Heuristics.  Journal  of  Pragmatics,  Vol.  59  (2013) 
210-220.  https://www.sciencedirect.com. 

Michta,  Andrew  A.  Polish  hard  power:  Investing  in  the  Military  as  Europe  Cuts  Back. 

(American  Enterprise  Institute,  December  19,  2013)  Available  online  from: 
http://www.aei.org/outlook/foreign-and-defense-policy/defense/nato/polish-hard-power- 

investing-in-the-military-as-europe-cuts-back/.  Accessed  on  6  April  2014. 

NATO  Standard  ATP-3.3.4.2.  Air-To-Air  Refueling  ATP-56,  Ed.  C  Ver.  1.  November  2013. 

NATO.  “Smart  Defense.”  Available  on-line  from: 

http://www.nato.int/cps/en/natolive/topics  84268.htm.  Accessed  on  18  Mar  2014. 

NATO.  “The  Connected  Forces  Initiative.”  Available  on-line  from: 

http://www.nato.int/cps/en/natolive/topics  98527.htm.  Accessed  on  18  Mar  2014. 

NATO.  NATO  Standardization  Agreement  4172:  5.56mm  Ammunition  (Linked  or  Otherwise),  Ed 
2,  Amd  2,  31  January  1995. 

Office  of  the  Undersecretary  of  Defense  (Comptroller).  Financial  Management  Reports, 
Department  of  Defense  FY2014  Reimbursable  Rates.  Available  on-line  from: 
http://comptroller.defense.gov/FinancialManagement/Reports/rates2014.aspx.  Accessed 
on  26  Mar  2014. 

OSD  Cost  Analysis  Improvement  Group  (CAIG).  Operating  &  Support  (O&S)  Cost-Estimating 
Guide.  Oct  2007.  Available  on-line  from:  https://acc.dau.mil/adl/en- 
US/188404/fde/32396/O  S  Cost  Estimating  Guide  Oct  2007.pdf  Accessed  on  24  Mar 
2014. 

Pellerin,  Cheryl.  DOD  Gives  High  Priority  to  Saving  Energy .  American  Forces  Press  Service. 
Available  on-line  from:  http://www.defense.gov/news/newsarticle.aspx?id=65480. 
Accessed  on  6  April  2014. 

Rasmussen,  Anders  Fogh.  Building  Security  in  an  Age  of  Austerity.  Speech:  2011  Munich 
Security  Conference  (February  2011).  Available  on-line  from: 
http://www.nato.int/cps/en/natolive/opinions  70400.htm.  Accessed  on  6  Apr  2014. 

Starosta,  Gabe.  “The  Air  Force’s  Fuel  Problem.”  Air  Force  Magazine,  July  2012.  Available  on¬ 
line  from: 

http://www.airforcemag.com/MagazineArchive/Pages/2012/Julv%202012/0712fuel.aspx. 

Accessed  on  27  Mar  2014. 

United  States  Transportation  Command.  Rates  and  Rules:  FY  14  Airlift  Rates.  Available  from: 
http://www.transcom.mil/rates/fV14Rates/.  Accessed  on  26  Mar  2014. 

Welsh,  Mark  A.  III.  Press  Briefing  by  Acting  Secretary  of  the  Air  Force  Eric  Fanning  and  Air 
Force  Chief  of  Staff  General  Mark  A.  Welsh  III  on  the  State  of  the  Air  Force  in  the 


Pentagon  Briefing  Room.  (13  Dec  2013). 

http  ://www.  defense .  gov/transcripts/transcript.  aspx?transcriptid=5 344 . 


